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Abstract 

About a decade ago, the Netherlands Environmental Assessment Agency 
(PBL) unwittingly embarked on a transition from a technocratic model of 
science advising to the paradigm of "post-normal science" (PNS). In 
response to a scandal around uncertainty management in 1999, a Guidance 
for "Uncertainty Assessment and Communication" v\^as developed with 
advice from the initiators of the PNS concept and was introduced in 
2003. This was followed in 2007 by a "Stakeholder Participation" Guidance. 
In this article, the authors provide a combined insider/outsider perspective 
on the transition process. The authors assess the extent to which the PNS 
paradigm has delivered new approaches in the agency's practice and analyze 
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two projects — on long-term options for Dutch sustainable development 
policy and for urban development policy — ^the latter in somewhat more 
detail. The authors identify several paradoxes PBL encounters when putting 
the PNS concept into practice. It is concluded that an openness to other 
styles of work than the technocratic model has become visible, but that the 
introduction of the PNS paradigm is still in its early stage. 
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post-normal science, technocracy, dealing with uncertainty, environmen- 
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Introduction 

On January 20, 1999, Dr. Hans de Kwaadsteniet, a senior statistician at 
the Netherlands Environmental Assessment Agency (PBL)' made news 
in the Netherlands. After years of trying to convince his superiors that the 
agency's environmental assessments leaned too much toward computer 
simulation at the expense of measurements, he went public with this criti- 
cism by pubhshing an opinion article in the national newspaper Trouw 
(de Kwaadsteniet 1999). His article was supplemented with an interview 
that resulted in the headline "Environmental Institute Lies and Deceives" 
on the newspaper's front page. His specific claim was that — ^being based 
on a virtual reality of computer models — the agency suggested too high 
an accuracy of the environmental figures published yearly in its State of the 
Environment report. 

Soon after the publication by de Kwaadsteniet, an intense and long-lasting 
media debate ensued in the Netherlands.^ The affair reached the floor of the 
Dutch Parliament within a matter of days. Facing the Parliament, the Minister 
of the Environment, Jan Pronk, defended the integrity of the agency. In return 
for an agreement that the agency would organize more regular external 
reviews of its activities and improve its communication of uncertainty, the 
Minister granted the institute additional funding for its monitoring activities. 
As is described in this article, the response of PBL to the affair was to intro- 
duce changes in its working style at the science-policy interface. 

The de Kwaadsteniet affair illustrates some of the difficult choices that 
have to be made by "boundary organizations" — ^that is, organizations that 
reside at the boundary between science and policy — in addressing very 
uncertain issues with high stakes involved. Should scientists communicate 
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about uncertainties with decision makers and let the decision makers decide 
on how to deal with these uncertainties or should scientists instead give in to 
the demand often expressed by decision makers to offer "certainties, " in the 
form of "best available scientific information" (thus, in effect repressing 
uncertainty)? Underlying the de Kwaadsteniet affair are different views 
on the role of science in policy. 

A traditional perspective on these views is offered by Yaron Ezrahi 
(1980), who discussed two opposing views: the " Utopian rationalist" and the 
"pragmatic rationalist" view. The "utopian rationalist" or "technocratic" 
ideal of science advice entails that the policy-making process should assim- 
ilate scientific information to a maximum extent. Utopian rationalism may be 
appropriate in those rare situations of consensus on both knowledge and val- 
ues that pertain to a policy problem. The technocratic ideal reflects the notion 
of science speaking "value-free" truth to political power that gained institu- 
tional currency in the nineteenth century. The notion of value-free science 
itself was based on the expectation that the impartiality and objectivity of 
scientists could help overcome political conflict (Proctor 1 99 1). In the twen- 
tieth century, however, it became clear both that science cannot be value-free 
and that politics deals increasingly often with issues that are clouded with 
uncertainty, including value diversity. The presence of conflict among scien- 
tists, both epistemic and social, makes it hard to provide politicians with neu- 
ttal advice. There is often considerable room for scientists and poUcy 
analysts to make choices in the assumptions of their analysis. 

Next to the "utopian rationalist" model, Ezrahi identified a "pragmatic 
rationalist," or "democratic" ideal of science advice that accepts, within 
limits, the inevitability of political ingredients in science advice. Pragmatic 
rationalism considers technocratic science advising to be mistaken for many 
policy problems. Uncertainty and value controversy ask for science to con- 
ttibute to political debate by representing different legitimate perspectives 
on policy problems. The approach is to recognize and identify the contro- 
versies, uncertainties, and ambiguities to open up discussion and stimulate 
the process of deliberative decision making. The post-normal science (PNS) 
paradigm that is studied in this article can be understood as a sfrategy for the 
production of relevant knowledge, in line with the "pragmatic rationalist" 
view on the science-policy interface. Fimtowicz and Ravetz (1993) intro- 
duced the term "post-normal science" for issue-driven knowledge produc- 
tion in a context of hard political pressure, values in dispute, high decision 
stakes, and high epistemological and ethical systems uncertainties. 

But how can this PNS paradigm be put into practice? The experiences of 
the Netherlands Environmental Assessment Agency may shed some light 
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on this question. After briefly summarizing the theory of PNS in the next 
section, this article describes how, since the de Kwaadsteniet affair — which 
caused a severe disruption in the agency's standard practice — ^the agency 
unwittingly embarked on a transition from the technocratic model of sci- 
ence advising to PNS paradigm. The seeds of the agency's transition to PNS 
were already present before or during the de Kwaadsteniet affair (Klo- 
progge and van der Sluijs 2006). Two emblematic projects in this regard 
were the TARGETS project (Rotmans and de Vries 1997) — ^which was cen- 
tered on a computer model that deals with value diversity and xmcertainty 
by allowing multiple model routes depending on a typology of so-called 
perspectives — and the COOL project (van de Kerkhof 2004) — a participa- 
tory Integrated Assessment project aimed at supporting development of 
long-term climate policy in the Netherlands and run by a consortium, in 
which the PBL participated to bring in its expertise. The agency has recently 
described itself as "an interface between science and policy," which is 
"independent" and aims to report "the full range of scientific opinions" 
(MNP 2006). How far the agency has come to incorporating the PNS para- 
digm is critically assessed in this article. 

The assessment is structured by referring to three key elements in the 
PNS paradigm: 

1 . The management of uncertainty: PNS acknowledges that uncertainty is 
more than a technical number or methodological issue. Ambiguous 
knowledge assumptions and ignorance give rise to epistemological 
uncertainty; 

2. The management of a plurality of perspectives within and without sci- 
ence: Complex problem solving requires scientific teamwork within an 
interdisciplinary group and joint efforts by specialists from the scien- 
tific community and from business, politics, and society; 

3. The internal and external extension of the peer community: An 
extended peer community includes representatives from social, politi- 
cal, and economic domains that openly discuss various dimensions of 
risks and their implications for all stakeholders. 

As will become evident to the reader, three of the authors of this article 
(AP, JvdS, and MH) have themselves been involved both in elaborating the 
theory of PNS and in the introduction of PNS in the practice of PBL. With 
advice from the initiators of the PNS concept, the agency introduced a Gui- 
dance for "Uncertainty Assessment and Communication" in 2003 and, with 
advice from political scientists, issued a "Stakeholder Participation" 
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Guidance in 2007. Tliese guidance documents offer methods and practical 
tools to incorporate the PNS approach in the agency's projects. Two of the 
authors of this article (EK and AC) were not directly involved with the ela- 
boration and introduction of these guidances; still, one of these (EK) later 
joined the agency and the other (AC) has acted as a consultant to the agency 
for the past few years. Thus, the article combines insider and outsider 
perspectives and it must thus largely be regarded as a self-evaluation. 

To illustrate the paradoxes encountered when the PNS paradigm is 
implemented into practice, the ideas behind the two guidances are described 
and two PEL projects are examined. A major example where especially the 
second element of the PNS approach — the plurality of perspectives — 
became clearly visible to the outside world, was the agency's "First Sus- 
tainability Outlook" (2004). The second example, examined in somewhat 
more detail, addresses in particular the third element and concerns a project 
on long-term options for urban development in the Netherlands which ran 
from 2008 to 2010. We end this article by discussing the paradoxes the PEL 
encoimters when putting the PNS concept into practice. 

The Paradigm of PNS 

The interface between science, policy, and society, or "science-policy- 
society interface" has been the topic of study of a whole body of political 
science and social studies of science literature (e.g., Jasanoff 1990; 
Fimtowicz and Ravetz 1993; Gibbons et al. 1994; Hisschemoller and Hoppe 
1996; Gieryn 1999; Guston 1999, 2000; Miller 2001; Nowotny, Scott, and 
Gibbons 2001; Collins and Evans 2002; Siebenhiiner 2004). Most of these 
studies have addressed questions pertaining to democratization and stake- 
holder participation. 

The science-policy-society interface is typically not regarded as a static 
given but instead can be seen as a highly dynamic process in which all kinds 
of linkages are continuously being formed and broken. The concept of 
"boundary work" (for an introduction see, e.g., Gieryn 1999; Jasanoff 
1990) is used by many studies in the literature to capture this dynamic. 
Eoundary work results in a demarcation and positioning of domains (for 
instance, science and policy or scientific disciplines) by defining who is 
seen as competent in specific areas of a policy problem. The boundaries 
between the domains of competence involved in a debate usually remain 
subject to continuous renegotiations. This boundary work is work done 
by all actors to both define and cross the boundaries between science, 
policy, and society. 
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Social scientists do not arrive at an equivocal view on the science — 
policy-society interface, neither on how it works nor on how it should 
work. In the current article, we focus on the normative theory of "post- 
normal science," which entails a reflective way of doing boundary 
work. The "post-normal science" paradigm (Funtowicz and Ravetz 
1993) has its roots in the mid-1980s. Recognizing that the technocratic 
interactions between science and policy making on controversial risks 
issues were often unproductive in cases where the decision stakes and 
system imcertainty are very high (notably in the case of nuclear energy), 
Sylvio Funtowicz and Jerry Ravetz proposed distinguishing a new type 
of risk assessment called "total-enviroimiental assessment" (Funtowicz 
and Ravetz 1985, 228). This is a form of risk assessment in which the 
"total envirormient" — that is, the complete context — of a risk issue is 
taken into account as much as possible. 

Similar to the risk assessment of nuclear energy in the 1980s, current day 
complex issues faced by the Netherlands Environmental Assessment 
Agency (PBL) are hard to tackle with normal scientific procedures. Activ- 
ities of the PBL, such as constructing environmental outlooks, sustainability 
outlooks, or assessing options for sustainable long-term urban development, 
involve deep uncertainties and value-ladenness, which cannot be dealt with 
in a "normal science" mode. The PNS paradigm is based on an acceptance 
of fimdamental uncertainty in science and policy and can be regarded as an 
initial impetus toward addressing the "complexity" of some pressing real- 
world problems. In comparison with the situation of "normal science," in 
which a linear view of the world prevails, PNS respects processes of emer- 
gence. Analysts within the PNS fi^amework know that the future is funda- 
mentally unpredictable. While for natural systems, this unpredictability 
can be mathematically explored using chaos theory involving laws of 
nature, social systems are even more fundamentally unpredictable since 
they involve human reflexivity. When we use the term "complexity" here, 
we mainly refer to the latter element of reflexivity. 

Where normal science aims at establishing the ultimate truth or the final 
resolution of a scientific puzzle, PNS recognizes that as long as both scien- 
tific uncertainties and decision stakes are high, such an aim is in principle 
unachievable. Indeed it can be misleading and create false expectations to 
act as if the role of science in such issues is just to get the facts right. 
Instead, PNS aims at common commitments to reflective approaches for 
dealing with complex policy issues. The PNS approach also allows room 
for "narratives" as a means of addressing complexity. Narratives provide 
important additions to factual claims, "because they are not about objective 
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reality, but are statements of what is significant" (Allen and Giampietro 
2006, 595). 

PNS is often misunderstood as something that replaces normal science. It 
should instead be seen as a societal problem-solving strategy that partly 
draws on normal science but emphasizes that for addressing complex policy 
problems more is needed to shape the science-policy-society interface. 
Facts are still necessary but no longer sufficient. The guiding principle of 
normal science — discovering the true facts — must be modified to fit the 
post-normal principle to achieve the goal of quality (Funtowicz and Ravetz 
1991). Traditional methods of quality assurance do not apply under those 
circumstances. Apart fi-om testing knowledge for validity and reliability, 
it should also be tested for "social robustness." The knowledge production 
is to be organized in a way that increases the social robustness and guaran- 
tees scientific quality at the same time. On this argument, Funtowicz and 
Ravetz (1994) defined the key elements of PNS. These include the appro- 
priate management of uncertainty, the plurality of perspectives and commit- 
ments, and the internal and external extension of the peer commimity. 

In the remainder of this article, we review the elaboration of the PNS 
paradigm in PBL's guidance documents and two projects to reflect on the 
transition of PBL from interfacing science and policy in normal "getting the 
facts right" mode toward the post-normal "working deliberatively within 
imperfections" (Funtowicz 2006) mode. 



Dealing with Uncertainty and Value Plurality 

P&L's Uncertainty Guidance 

The govemment-fimded Netherlands Environmental Assessment Agency 
(PBL) supports national and international policy makers by providing inte- 
grated assessments on topics such as sustainable development, energy and 
climate change, biodiversity, transport, land use, air quality, and spatial 
planning. After the crisis described in the beginning of this article, the PBL, 
or, more accurately, its immediate predecessor within RIVM, was reviewed 
by an international expert committee in 2000, which recommended to start a 
project to "systematically address terminology, methodology, interpreta- 
tion and communication of uncertainty" (RIVM 2000). A subsequent 
review at the national level led to the proposal to develop a protocol or 
guideline (van Asselt et al. 2001). 

In 2001, a project was started at PBL to develop, in collaboration with 
Utrecht University and involving an international team of imcertainty 
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experts, a Guidance for Uncertainty Assessment and Communication 
(Petersen et al. 2003; Janssen et al. 2003; van der Sluijs et al. 2003; van der 
Sluijs et al. 2004; Petersen 2006; van der Sluijs et al. 2008). In the project, 
an early choice was made to de-emphasize (though not neglect) two domi- 
nant understandings of how to deal with uncertainty at the science-policy- 
society interface — the "deficit view" and the "evidence evaluation view" 
(explained below) — and to focus more on the further development of the 
"post-normal view" (van der Sluijs 2006). 

Within the deficit view, uncertainty is considered a deficit of available 
knowledge. Uncertainty is seen as a temporary problem that will disappear 
automatically if more research would be performed. In this view, manage- 
ment of uncertainty equals reduction of uncertainty and there is a strong 
belief that science is ultimately able to provide certainty. Where uncertainty 
can not yet be reduced, uncertainty is treated as if it were a fact (usually a 
percentage number that represents for instance two standard deviations over 
a mean value) that needs to be discovered and correctly quantified. In 
monodisciplinary domains there is a whole science on how to calculate the 
"true" uncertainty in for instance measurement data (e.g., Taylor 1982). 
One tendency typically seen in the deficit view is the production of ever 
more complex and detailed models, because precise calculation is seen as 
key to truth. The techniques applied include Monte Carlo, Bayesian beUef 
networks, and other quantification techniques. A pitfall of this paradigm can 
be that a false certainty is created, because the numbers obtained fi-om these 
models can suggest more knowledge than there actually is. The deficit view 
is often encountered in monodisciplinary normal science. At PBL, the def- 
icit view was especially prevalent among modelers and economists. 

The evidence evaluation view considers uncertainty to be a problematic 
lack of equivocalness. When science speaks with multiple voices to policy, 
conflicting certainties may emerge. The solution proposed is a comparative 
evaluation of individual research results, focused on building scientific con- 
sensus. The focus shifts from establishing certainty to evaluation of evi- 
dence to establish gradations of certainty. Multidisciplinary expert panels 
such as the Intergovernmental Panel on Climate Change (IPCC) have been 
established for this purpose. This view focuses on generating robust conclu- 
sions and widely shared interpretations of the available limited knowledge. 
A pitfall of this paradigm can be that matters on which no consensus can be 
reached continue to receive too little attention, while, in fact, this dissension 
is often highly policy-relevant. Especially weak signals of early warning of 
(new) risks are likely to be overlooked or underaddressed here, because it 
often is impossible to reach a consensus interpretation on such issues. The 
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evidence evaluation view was recognizable in PBL's multidisciplinary 
assessments of complex policy problems. 

The post-normal view that was farther developed in PBL's Uncertainty 
Guidance sees uncertainty as intrinsic to complex systems and thus as a per- 
manent phenomenon. In farther contrast with the deficit view, it sees uncer- 
tainty as more than a number. It stresses that uncertainty also results from 
the new ways by which knowledge on complex policy issues is produced. 
For instance, the use of computer simulation models, scenarios, and extra- 
polations all critically depend on the validity of the assumptions that una- 
voidably need to be made. Most of such assumptions can in principle not 
be validated (see also Konikow and Bredehoeft 1992; Oreskes, Shrader- 
Frechette, and Belitz 1994). This implies that if knowledge is produced that 
is conditioned on (the unknown validity of) assumptions, uncertainty is una- 
voidably coproduced. The post-normal view acknowledges that not all 
uncertainties can be quantified and that in complex issues, unquantifiable 
uncertainties can well be more relevant and salient than the part of uncer- 
tainty for which we have enough knowledge to quantify it in some reliable 
way. It calls for an approach that openly deals with deeper dimensions of 
uncertainty, such as those stemming from problem framing, choice of sys- 
tem boundaries, indeterminacy, ignorance, assimiptions, value loadings, 
imderdetermination (the same data allowing for several interpretations and 
conclusions), and even institational dimensions. 

The Uncertainty Guidance proposes a more qualitative and reflective 
approach to uncertainty (van der Sluijs et al. 2008). Techniques that are 
applied to deal with this are Knowledge Quality Assessment tools such as the 
NUSAP system (van der Sluijs et al. 2005) and risk management (including 
production of knowledge) as a deliberative (participative) social process. A 
pitfall of the PNS paradigm is that uncertainty can be highlighted to such an 
extent that we may forget how much we acfaally do know about the risk con- 
cerned and on which aspects there is, in fact, consensus. Within the Gui- 
dance, the PNS paradigm therefore complements and does not replace the 
consensus approach and the uncertainty quantification approach. 

In the post-normal domain, the traditional forms of scientific quality 
control such as peer review and validation run into their limitations. For 
instance, if scientific insights required for decision problems cannot be pro- 
duced without computer simulation models, the resulting knowledge caimot 
be validated (Oreskes, Shrader-Frechette, and Belitz 1994). To maintain 
quality, a modeler has to be a good craftsperson (Ravetz 1971). In cooper- 
ation with PBL, Utrecht University developed several Knowledge Quality 
Assessment tools to assist modelers. Discipline can be maintained by 
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controlling the introduction of assumptions into the model and maintaining 
standards of "good practice." A new method was developed to identify, 
review, and prioritize assumptions to assess the potential value-ladenness 
of important assumptions and to deal with these potentially value-laden 
assumptions in an explicit and transparent manner (Kloprogge, van der 
Sluijs, and Petersen 2011). Furthermore, Risbey et al. (2005) developed 
a heuristic that encourages self-evaluative systematization and reflexivity 
on pitfalls. It takes a reflective approach to the management of quality by 
providing guidance on good modeling practices and at the same time pro- 
viding diagnostic help as to where problems may occur and why. It recog- 
nizes that models are tools, not truths. A model is not good or bad but there 
are "better" and "worse" forms of modeling practice. A crucial notion is 
that models are "more" or "less" useful when applied to a particular 
problem. Reflective approaches to quality can thus provide insurance 
against pitfalls in process and insurance against irrelevance in application. 
Risbey et al. (2005) applied their approach to PBL's global energy 
model TIMER. 

The Uncertainty Guidance distinguishes six parts of enviroimiental 
assessments that merit separate attention: (1) problem framing; (2) involve- 
ment of stakeholders; (3) selection of indicators; (4) appraisal of the knowl- 
edge base; (5) mapping and assessment of relevant uncertainties; and (6) 
reporting of the uncertainty information. The Uncertainty Guidance tackles 
all three key elements of PNS theory mentioned in the introduction: the 
appropriate management of uncertainty and of quality; the plurality of per- 
spectives; and the extension of the peer community. The Guidance is set up 
as a checklist with pointers to hints and actions of good practice. 

In the view on the science-policy-society interface that underlies the 
Uncertainty Guidance, the shaping of this interface is strongly related to the 
framing of the policy problem. In the first step of the guidance, the problem 
and its context and history are outlined, by identifying major issues, past 
work, the level of contention, and the (expected) role of the assessment 
in the policy or decision-making process. The assessor is explicitly asked 
to consider various views/perspectives on the problem and to pay attention 
to the problem's interconnectedness with other problems. She or he is asked 
to be specific on what knowledge is needed with regard to the problem and 
into which research questions this is translated. Possibly relevant aspects 
that are not dealt with in these research questions have to be indicated. 
Moreover, it should be outlined what role the study is expected to play in 
the policy process and what the relation is with previous studies on the 
subject (policy context and problem history). 
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Although the Uncertainty Guidance is not Mly used within all PBL 
projects yet, it is increasingly used, attitudes toward dealing with uncer- 
tainty in performing and reporting environmental assessments — the first 
PNS element — have changed, and communication on uncertainty in PBL 
reports has improved over this period (Wardekker et al. 2008). Below we 
will briefly illustrate how also the second PNS element — dealing with value 
plurality — ^has received a more prominent role in PBL. 



PBL's First Sustainability Outlook 

The Uncertainty Guidance's advice on explicitly addressing different per- 
spectives was most visibly put into practice in an important agency project 
that was concluded in the year after the guidance was introduced: the First 
Sustainability Outlook (MNP 2004; de Vries and Petersen 2009). In this 
project, after much heated internal debate on how global sustainable devel- 
opment should be conceptualized and operationalized for a country such as 
The Netherlands, a method was developed to explicitly deal with divergent 
perspectives on the problem, which built on the experiences gained by the 
agency in the TARGETS project. Such explicit treatment of underlying 
value commitments proved to be a radical break with the way environmen- 
tal assessment were conducted previously in the agency. 

Starting point of the method is that a sustainability assessment should 
investigate the ability to continue and develop a desirable way of living 
vis-a-vis later generations and life elsewhere on the planet. Evidently, peo- 
ple hold different values and beliefs about the way societies sustain quality 
of life for their members. The first step, therefore, is to analyze people's 
value orientations and the way in which they interpret sustainability prob- 
lems, that is, their beliefs. The next step is to translate the resulting world- 
views into model-based narratives, that is, scenarios. The qualitative and 
quantitative outcomes are then investigated in terms of associated risks and 
opportunities and robustness of policy options. This method was applied, 
using extensive surveys among the Dutch population, in the First Sustain- 
ability Outlook. The resulting archetypical worldviews became the basis for 
four different scenarios for policy analysis, with emphases on the domains 
of transport, energy, and food. The goal of the agency's Sustainability Out- 
looks is to show that choices are inevitable in policy making for sustainable 
development, to indicate which positive and negative impacts one can 
expect of these choices (trade-offs), and to identify options that may be 
robust under several worldviews. 
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The use of different worldviews provides a way to deal with the plurality 
of perspectives on sustainability problems. Obviously, for a policy assess- 
ment on sustainable development to be able to play a significant role in 
structuring the policy debate, stakeholders need to be engaged and feel rep- 
resented in the worldviews used. In this project, the level of participation 
was still kept relatively low. Scenario experts were consulted on the sce- 
nario methodology and consultative sessions were held at four ministries, 
the social-economic council, a government strategists group, a high-level 
environmental network organization, a group of university professors and 
three nongovernmental organizations (NGOs). Still, the methodology of 
worldviews as a tool for strategic policy making proved to be helpful in pol- 
icy exercises that were performed after the assessment had been published. 

Dealing with an Extended Peer Community 

PBL's Stakeholder Participation Guidance 

In the post-normal domain, scientific and technical discourse is no longer 
restricted to expert communities but needs to be inclusive of nonspecialist 
participants (stakeholders and citizens; Funtowicz and Ravetz 1993). This 
requires the involvement of an extended peer community and participation 
of stakeholders and citizens not only in the phase where solutions are debated 
but also in the assessment process that precedes it. Since no particular exper- 
tise can deliver certainty for policy issues in the post-normal domain, no 
expertise can claim a monopoly of wisdom and competence. Indeed, relevant 
wisdom is not limited to scientific speciaHsts and public officials. 

Stakeholders can contribute to knowledge in a number of ways: by co- 
framing the problems addressed by the scientists (e.g., what are the relevant 
aspects of a problem that need be included), by acting as a source of non- 
scientific knowledge, and as a source of critical reflection in quality control 
of the scientists' work (e.g., by challenging the validity of assumptions 
made in assessment studies). 

Thus, PNS requires a dialogue based on the recognition of a plurality of 
legitimate perspectives and areas of expertise vis-a-vis the problem, each 
with its own contributions of relevant evidence, commitments, and insights. 
Its goal should not be to establish which single voice is "right," inevitably 
making the others "wrong" (Fimtowicz and Ravetz 1994). Instead, a rich 
and diverse body of (sometimes conflicting) potentially policy-relevant 
evidence is coproduced and matures by criticism across the perspectives 
involved in the inclusive dialogue. 
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This PNS mode of interfacing science and policy can be summarized as 
"working deliberatively within imperfections" (Funtowicz 2006; 
Fnntowicz and Strand 2007; van der Sluijs and Funtowicz 2008). This 
requires new interdiscipHnary contacts and integration (internal extension 
of the peer community) on one hand, and new "knowledge partnerships" 
with policy makers, NGOs, industry, media, and the public (external exten- 
sion of the peer community) on the other hand, to meet the challenges of 
quality control in the assessment of complex risks. 

After the Guidance for Uncertainty Assessment and Communication was 
published in 2003, the Netherlands Environmental Assessment Agency 
decided to add a more detailed Stakeholder Participation Guidance (Hage 
and Leroy 2008; Hage, Leroy and Petersen 2010) to its toolkit for environ- 
mental assessors, to facilitate both types of peer community extension. It is 
organized around a number of guiding questions: 

1 . Why do you want participation? 

2. What should the participation be about? 

3. Who do you want to involve? 

4. How much participation do you want? 

5. What form are you choosing? 

One of the most important things when organizing participation is to formu- 
late clear aims. The Netherlands Environmental Assessment Agency does 
not consider stakeholder participation in science-for-policy as an aim in 
itself for each project. But for cases in which stakeholder participation is 
found important, the Stakeholder Participation Guidance offers advice on 
how to organize it. 

The "Sustainable City" project, which explores long-term options for 
urban development in the Netherlands, is the first "PBL-only" project in 
which stakeholder participation plays a central role. It started in 2008 for 
a duration of two years. Below we present some insights into the relation 
of the project to PNS practices (addressing not only the extension of the 
peer commimity but also the other two key elements of the PNS paradigm: 
managing uncertamty and a plurality of perspectives). 

PB)L's Sustainable City Project 

The aim of PBL's Sustainable City Project was to generate integrated 
options for long-term urban development pohcies in the Netherlands. The 
project started from the assumption that today's poHcy on urban 
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development is too fragmented and not well considered concerning 
side-effects and trade-offs. By integrating different scientific disciplines 
and different kinds of knowledge, the project tries to identify those policy 
options that will trigger sustainable urban development. The central ele- 
ment of the project consists of stakeholder dialogues concerning three main 
topics: health, livability, and energy. The procedure was modeled to the par- 
ticipatory backcasting approach (see Quist and Vergragt 2006; Carlsson- 
Kanyama et al. 2003; Carlsson-Kanyama et al. 2008). The primary idea 
of the project is to ask stakeholders from different fimctional areas (health, 
energy, and livability) to deliver their ideas about the "sustainable city" in 
the year 2040, within their field of expertise. The project team made the 
choice to start from three different perspectives to identify synergies and 
ttade-offs between these three aspects of sustainability. 

The choice for the participatory approach was triggered by the growing 
awareness at the agency that "wicked problems" such as sustainable devel- 
opment need another methodological approach than the traditional ones. 
The project aimed at method development to address this type of problems, 
using an analytic-deliberative approach to address sustainability in an 
urban context. An analytic-deliberative approach offers an opportunity for 
creating "a decision framework that stresses not only technical information, 
but also the explicit input of values, insights and tradeoffs" (Petts 2004, 
116), which can be obtained with stakeholder involvement. At the same 
time, an overall philosophical discussion about the conceptualization of 
"quality of life" emerged at the agency after the publication of the First 
Sustainability Outlook (for a brief description, see previous section). A 
highly abstract definition of quality of life was considered as not usefiil for 
this project and it was decided to take a "hands-on" approach to sustain- 
ability in the urban context. Knowledge about the important issues for urban 
areas was judged as poor, because there are hardly scientific models that can 
assess sustainability at a local level. Also for this reason, the project leader 
decided to involve stakeholders; to improve the knowledge base by asking 
the stakeholders to bring in their knowledge of the urban context. 

Three rounds of workshop were conducted. In the first workshop, stake- 
holders were asked for their ideas to make an imaginary city respectively 
more healthy, livable, and energy-neutral and to enhance the quality of life. 
The group of stakeholders consists of practitioners, (local) policy makers, 
architects and builders, NGO representatives, and scientists. In between the 
two workshops, the stakeholders were given the opportunity to elaborate 
their statements made in the workshop by participating in an internet- 
facilitated discussion. In the second workshop, the stakeholders were asked 
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to identify pathways to the future, using the participatory backcasting 
method. 

The project team assessed the values, beliefs, and knowledge claims on 
the future orientation of sustainable urban life and their implications for 
spatial planning. 

A mixed assessment approach is applied consisting of scenario analysis 
and narrative analysis. Narrative analysis served as input to development of 
the policy scenarios. Scenario analysis was done by means of urban systems 
simulation modeling and impact assessment on health, livability, and 
energy. A selected set of stakeholders was invited to peer review the anal- 
ysis process and its outcomes, paying particular attention to the modeling 
assumptions. Narrative analysis offers insight into normative and institu- 
tional perspectives on sustainable city planning. The stakeholders support 
that sustainable development at the urban level asks for a transition to a 
different governance structure, where new and diverse alliances of local 
citizens, business, and organizations are responsible as producers and con- 
sumers of societal value. The role of the national government should be to 
provide clarity and consistency by offering a challenging and inclusive 
future vision, while implementation is bounded to local and regional levels 
where economic, ecological, and social sustainability can enhance and 
strengthen one another. 

At the third, final workshop the assessment findings were demonstrated 
on the basis of the model outcomes of the scenario analysis. The question 
was addressed of how the national government can support local govern- 
ment and private parties in developing sustainable urban environments, 
knowing that existing structures and positions will have to change into new 
alliances. 

Can the project "Sustainable City" be classified as post-normal? As 
probably most of sustainability research projects, the "Sustainable City" 
project contains post-normal elements but is actually a hybrid package of 
different kinds of topics with different kinds of uncertainties and different 
degrees of value-ladenness. The project design gives an important role to 
stakeholder participation to enhance the quality of knowledge by bringing 
scientists and practitioners together and making use of practical knowledge. 
However, during the design and assessment process of the project, the proj- 
ect team had to face different dilemmas that revealed a tension between 
post-normal and normal science. This is most obviously expressed by not- 
ing that the project members only realized later in the project how important 
the narrative elements of the stakeholder dialogue actually were for recon- 
ciliation of the deliberative and analytic processes. Using rigorous methods 
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such as scenario design and system-bounded models only partially allowed 
for incorporation and processing of the creativity and out-of-the box ideas 
that were generated by the stakeholders. This reductionist approach is inher- 
ent to the use of such methods. The seeds of opening up this project to a 
more PNS-like approach were present in PBL's organizational adoption 
of the Uncertainty and Stakeholder Participation Guidances and was also 
due the participation of two of the authors (MH and EK) within the project 
team to help guide the project toward more openness. The process of open- 
ing up is illustrated with reflective fragments of the project in the following 
paragraphs. 

Framing of knowledge. One of the central questions was how far one could 
and should control the stakeholder interaction for the benefit of the projects' 
research question. The initial idea was to give the stakeholders the same 
clear-cut definition of the issue at stake as the definitions the models are 
based on. This should also avoid unnecessary discussions and create a com- 
mon starting point of debate. However, such a restriction would do no good 
for the motivation of the participants and the dynamics of the workshops. 
That is the reason why the project team decided to give the stakeholders the 
opportunity to co-frame the issue at stake. The first workshop endorsed this 
expectation: the participations did not want to discuss about criteria and 
cause-effect relationships but did want to express their perspective about 
what the "good life" could look like. An example from the first health 
workshop can illustrate this: the discussion shifted from definitions of 
health and effectiveness and cause-effect relationships of health- 
improving measures in urban design to quality of life in general, happiness, 
and social interaction. 

Use of stakeholder knowledge. A major challenge of the project was to 
realize a dialogue between the (mostly quantified) scientific knowledge 
of the project team (the models used for health, energy, and livability) on 
one hand, and the practical knowledge of the stakeholders on the other 
hand. At the beginning of the project, stakeholders were considered as a 
source of knowledge that can be used like other sources of quantifiable 
information. But it turned out that the initially expected role of the stake- 
holders — ^to "feed" the models of the scientists involved with 
quantifiable information — was hard for them to fulfill. If the project team 
reduced the contribution of the stakeholders to the limited parameters of 
the models, a lot of valuable information would have been lost. For this 
reason, the project design opened up to a more narrative approach, allowing 
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for a debate on what the "good hfe" could be instead of prescribing such 
concepts from a scientific perspective. 

Scale of knowledge. Another methodological problem of the project was 
the tension between generalized and contextual knowledge on the topic 
of urban development. The Netherlands Enviroimiental Assessment 
Agency reports to the national government. The knowledge production pro- 
cess of the "Sustainable City" project is therefore oriented to produce 
knowledge that facilitates sustainable urban development on a national 
policy level. This should be generalized knowledge, independent from a 
particular local context. That is why the project team chose to ask the sta- 
keholders to create ftiture visions of an imaginary city and not an existing 
one. The project team was afraid of getting a discussion on "pavement 
level" concerning a local context that would not deliver useful information. 
However, contextual knowledge is essential for obtaining insight into pol- 
icy processes and feasibility of strategies. Additional case study research 
would probably have delivered valuable insights into the feasibility of the 
options for urban development. The tension perceived in this project illus- 
ttates the general problem of the agency to make use of case study methods. 
The agency is respected to produce context-independent knowledge that is 
generalizable at a macro level. Knowledge produced by qualitative empiri- 
cal social research does seldom meet this expectation. 

Stakeholder selection. The selection of the stakeholders took place from an 
expert perspective. Participants should deliver ideas and policy options 
from their field of expertise. They should also be able to think on a higher 
level of absfraction to deliver context-independent knowledge. This 
resulted in a selection where participants were mostly highly educated, high 
income class, and white. Considering the value-ladenness of the issues at 
stake, other criteria of stakeholder selection with a better representation 
of the different stakes and societal perspectives on the whole might have 
been more appropriate. This could have strengthened the knowledge base 
and the quality of the dialogue in terms of social robustness. 

When we evaluate this ongoing project, we see signs of a shift from 
technical and analytic research to analytic-deliberative research where 
model-driven knowledge is reconciled with stakeholder knowledge, using 
quantitative and qualitative approaches in liaison. As a first conclusion, 
we see that managing plural perspectives and opening up to "extended peer 
review" requires less a revision of the ttaditional scientific knowledge than 
a shift in focus of the research design: the Sustainable City project opened 
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up to a more narrative approach. As a consequence, however, the project 
suffered from a lack of focus. A lesson learnt from this project is that 
involving an extended peer community into a debate on wicked problems 
requires more intensive framing to allow for in-depth analysis of the various 
problem perspectives. 

As a second, more general conclusion, one might say that this project 
demonstrates how PBL is seeking to reposition itself into a more post- 
normal reality while in a way maintaining the status of a "normal" scientist 
who is able to produce generally valid long-term options for urban develop- 
ment in the Netherlands. 

Discussion and Conclusion 

What we have learned over the past few years is that the institutional chal- 
lenges of implementing practices of PNS, for example, through "gui- 
dances," in a government agency with several hundreds of employees 
should not be underestimated. It entails much more than disseminating the 
documents through an organization. For example, PBL's top management 
has ordered and subsequently endorsed the guidances; PBL's methodology 
group led the development of the guidance; the use of this guidance is now 
mandatory as part of the agency's quality assurance procedures; and the 
staff is actively trained to acquire the necessary skills. They are slowly, but 
steadily, starting to adopt or more PNS-like mind-set. In addition, a meth- 
odological support unit is available in the agency to assist and advise in 
assessment projects. The required process of cultural change within the 
institute was consciously managed over the whole period of 2003 until now. 

One requisite for successfully implementing the PNS-based guidances is 
that the agency staff will actually have to understand and share the basic 
underlying theory. It is not obvious that this is already fully the case. Some 
evidence for obstacles to dealing with a plurality of perspectives — in partic- 
ular, a plurality of normative views — was found, for instance, in a socratic 
conversation held with a small group of crucial staff members (Petersen, 
Melse, and Cath 2006). Three questions were addressed in this 
conversation. 

The first question posed to the agency staff was "Under what conditions 
can normative perspective plurality be made finitfiil for the staff?" There 
are many ways to deal with normative perspective plurality, with both pros 
and cons. And there are different levels at what such complexity can be 
dealt with (individual, project, and organization). According to the group, 
the important issue to discuss in this regard was how different perspectives 



380 



Science, Technology, & Human Values 36(3) 



are represented within the agency as compared with the perspectives that 
are held outside the organization. Three positions were taken: (1) some 
claimed that the population of employees should be diversified to include 
more people from the political right; (2) others thought that that cannot 
be realized and that instead different roles should be played (you do not 
need to be conservative to act like one), which should be considered as a 
competence of the professional; and (3) again others disagreed with both 
previous positions and claimed that the real problem is not a lack of differ- 
ent perspectives, but an abundance of pluriformity, which needs to be dealt 
with. 

A second question followed up on the previous discussion: "Suppose 
that there is a dominant frame within the agency; can this be remedied by 
playing roles?" Some participants recognized that the agency staff perform 
their analyses from a relatively narrow frame. In how far there is room for 
perspective plurality and dealing with real complexity is a matter that 
touches upon the mission of the organization? Is it necessary or useful for 
policy evaluations to consider all complexity and perspective plurality? Is 
the "post-normal science" method needed for all products that the agency 
delivers? Others gave examples of playing roles for the purpose of their reg- 
ular policy evaluation tasks: for example, a Lower House session was orga- 
nized about air quality, which did produce new insights ("We all agreed 
with each other and with discussion we did not get any step further within 
the agency, so we took different roles."). Such a role play would also have 
been usefiil in other cases is noted by participants. Still, a few of the parti- 
cipants would very much like to see the director to make a definite choice 
after having heard the different viewpoints. Although the majority of people 
in the group thought it is necessary, there was still some reminiscent fear to 
communicate externally about different perspectives and uncertainties con- 
cerning environmental problems. 

The discussion was therefore concluded with a third question: "Shall we 
put different perspectives next to each other and stay neutral, or shall we 
make a choice for one position, or — third alternative — are we pragmatic 
and does it not really matter where we end in dealing with perspective plur- 
ality?" The participants preferred to stimulate plurality and note that from 
internal discussions new insights can emerge and creative solutions can be 
found. Actually, the policy makers themselves will have to decide under 
uncertainty and they may appreciate us for giving them uncertainty infor- 
mation ("One of our products is to offer pluriform assessments"). It was 
concluded that dealing with pluriformity and complexity should be done 
when it is necessary, but that it cannot be put in a "protocol." 
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The current situation at the Netherlands Environmental Assessment 
Agency manifests a number of paradoxes. While some of the agency staff 
claim that they work according to a PNS approach, the actual mode of 
working of most of the agency staff has only slowly begun moving in the 
direction of the PNS paradigm. In the First Sustainability Outlook, for 
instance, hardly any stakeholder was involved — four archetypical perspec- 
tives were formulated, which were not derived in a bottom-up process or 
tested for their representativeness and usability. Still the presentation of 
value-laden aspects of a problem that could also have been treated in a more 
"objective" manner constitutes progress in the direction of PNS. And the 
"Sustainable City" project did open up to a post-normal strategy, even 
though some members of the project team still aimed for "normal" out- 
comes vwth assumed general validity for national policy making. Further- 
more, given the institutionalized role of the Netherlands Environmental 
Assessment Agency at the Dutch science-policy interface and regular reor- 
ganizations (the latest due to a merger), the modest progress made in the 
direction of a PNS strategy should be considered a substantial result. It is 
not clear how much further the agency could go even, without losing some 
of its credibility in the policy domain (based on the image of "normal sci- 
ence"). In February 2010, the agency was charged with evaluating the qual- 
ity of the Working Group II contribution to the Fourth Assessment Report 
by the IPCC in 2007. PBL approached this task by performing a limited 
evaluation within the PNS paradigm. PBL's scientifically educated policy 
analysts, who are not experts in climate-impacts science, represented the 
extended peer community, as well as several interested Dutch individuals 
who submitted comments on the IPCC report to PBL's public registration 
Web site. Through its evaluation, published in July 2010 (Meyer and 
Petersen 2010), PBL reflected on the qualitative judgments and value com- 
mitments that necessarily permeate the IPCC assessment process. The PNS 
approach taken by PBL to evaluating the IPCC report — ^by critically focus- 
ing on issues of transparency as outsiders — was found risky vis-a-vis PBL's 
own credibility. The IPCC could have publicly denounced PBL as not being 
sufficiently competent in the area of climate-change impacts (a "normal 
science" criticism of the evaluation report). Until policy expectations 
toward the role of PBL ask for socially robust knowledge in addition to the 
scientific rationale, possibly other institutions are better suited to take the 
PNS approach to its extremes. 

Even the modest progress toward PNS observed for the agency involved 
a considerable implementation effort. The new strategy for dealing with 
uncertainty and stakeholders has been introduced mainly by way of a 
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training in the order of two workdays, but this initiation in PNS paradigm is 
not sufficient to reach the level of in-dwelling (cf. Polanyi 1958) needed for 
thorough understanding. A change in working procedures is a slow process 
that naturally is met with resistance. Change is facilitated by external fac- 
tors (such as societal expectations toward PBL) and internal factors such 
as availability and acquaintance with new practices for knowledge produc- 
tion. The training is focused on recognizing and acknowledging uncer- 
tainty, and as such can be considered useful. However, for the agency to 
fulfill its self-proclaimed "knowledge brokerage" function (the agency 
calls itself publicly a "bridge" between science and policy), it runs into the 
problem of an effective maintenance of multidisciplinarity and complexity. 

Knowledge that is discoimected fi-om values, circumstances, and con- 
crete persons is in danger to reduce complexity to the extreme. Multidisci- 
plinarity does not just require an ability to see from more than one point of 
view. It has to provide an ability to let the different perspectives (in science, 
policy, and society) commimicate with one another, influence one another, 
and reach some sort of coherence. However, by emphasizing "science" and 
denouncing "postmodernism" (cf. Funtowicz and Ravetz 1985), there is the 
risk in the PNS paradigm of accepting the truth claims of what the stake- 
holders or practitioners acknowledge and take for real. The risk of self- 
reproducing truth claims — in particular with respect to social reality — is a 
clear and present danger when practicing PNS in a context of normal sci- 
ence expectations. The methodological challenge ahead, not only for the 
organization studied in this article, is to find answers to the question how 
can we reach understanding which addresses complexity even deeper. As 
said, it remains questionable whether an agency such as PBL can go even 
further than the PNS paradigm, given its institutional context and the obsta- 
cles that are already encountered in moving toward the PNS paradigm. But 
even though the introduction of the PNS paradigm is still in its early stage in 
PBL, we can conclude that an opeimess to other styles of work than the 
technocratic model has become visible in PBL's practice. 
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Notes 

1. The Netherlands Environmental Assessment Agency is a governmental body that 
by statute provides the Dutch government and parliament — and, in addition, the 
European Commission, European Parliament, and UN organizations — ^with sci- 
entific advice on environmental, sustainability, and spatial planning problems. Its 
acronym has changed over the past decade from RIVM (Rijksinstituut voor 
Volksgezondheid en Milieu, of which the agency's function was a part), through 
MNP (Milieu- en Natuurplanbureau, since 2003), to PBL (Planbureau voor de 
Leefomgeving, the Dutch name since May 2008, when it merged with assess- 
ment agency RPB, to include assessments of spatial planning issues). The 
English name has remained unchanged since 2003. 

2. See van Asselt (2000) and van der Sluijs (2002) for more information about this 
debate. 
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